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NEURAL  MECHANISMS  FOR  COORDINATE  TRANSFORMATIONS 


R.  A.  ANDERSEN.  P.  MAZZONI.  &  M.  JORDAN 
Massachusetts  Institute  of  Technology 


Area  7a  of  the  posterior  parietal  cortex  of  the  primate  brain  is  concerned  with 
representing  head-centered  space  by  combining  information  about  the  retinal 
location  of  a  visual  stimulus  and  the  position  of  the  eyes  in  the  orbits.  We  previously 
trained  a  neural  network  model  of  area  7a  to  perform  this  coordinate  transformation 
task  using  the  back-propagation  training  procedure.  The  hidden  units  of  this 
network  developed  properties  very  similar  to  those  of  area  7a  neurons  presumed  to 
code  for  spatial  location. 

Here  we  describe  two  neural  networks  with  architecture  similar  to  that  of  our 
previous  model  and  trained  them  to  perform  the  same  coordinate  transformation  task. 
However,  the  networks  were  trained  with  a  version  of  the  "associative  reward 

penalty"  (Ar-p)  algorithm  that  we  modified  to  work  on  a  multi-layered  network.  This 

learning  procedure  was  devised  by  Barto  and  colleagues,  and  is  more  biologically 

plausible  than  back-propagation.  Our  networks  learn  to  perform  the  task 
successfully  to  any  accuracy  and  almost  as  quickly  as  the  original  model,  and  the 
hidden  units  develop  response  properties  very  similar  to  those  of  area  7a  neurons. 

We  found  that  the  synaptic  strengths  computed  by  the  (Ar-p)  algorithm  are 
equivalent  to  and  interchangeable  with  those  computed  by  back-propagation.  Our 
networks  also  acquire  interesting  generalization  properties,  that  is,  they  learn  to 

perform  the  correct  transformation  on  pairs  of  eye  and  retinal  positions  never 
encountered  before.  All  these  results  are  unaffected  by  the  interposition  of  an  extra 
layer  of  units  between  the  hidden  and  output  layers. 

These  results  show  that  the  response  properties  of  the  hidden  units  of  a  layered 
network  trained  to  perform  coordinate  transformations,  and  their  similarity  with 
those  of  area  7a  neurons,  are  not  a  specific  result  of  back-propagation  training. 

They  seem  rather  to  derive  from  the  parallel  architecture  of  the  model  network  and 
from  the  principles  involved  in  learning  by  examples.  The  fact  that  they  can  be 
obtained  by  a  more  biologically  plausible  learning  paradigm  corroborates  the 
validity  of  this  type  of  neural  network  structure  as  a  plausible  model  of  how  area  7a 
may  compute  coordinate  transformations. 
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ANIMATE  VISION 


DANA  BALLARD 
University  of  Rochester 


Animate  vision  systems  have  gaze  control  mechanisms  than  can  actively  position 
the  camera  coordinate  system  in  response  to  physical  stimuli.  Compared  to  passive 
systems,  animate  systems  show  that  visual  computation  can  be  vastly  less  expensive 
when  considered  in  the  larger  context  of  behavior. 

The  most  important  visual  behavior  is  the  ability  to  control  the  direction  of  gaze. 
This  allows  the  use  of  very  low  resolution  imaging  that  has  a  high  virtual  resolution. 
Using  such  a  system  in  a  controlled  way  provides  additional  constraints  that 
dramatically  simplify  the  computations  of  early  vision.  Another  important  behavior 
is  the  way  the  environment  "behaves."  Animate  systems  under  real-time  constraints 
can  further  reduce  their  computational  burden  by  using  environmental  cues  that 
are  perspicuous  in  the  local  context.  A  third  source  of  economy  is  introduced  when 
behaviors  are  learned.  Because  errors  are  rarely  fatal,  learning  algorithms  can 
amortize  computational  cost  over  extended  periods.  Further  economies  can  be 
achieved  if  the  learning  system  uses  indexical  reference,  which  is  a  form  of  dynamic 
variable  binding.  Animate  vision  is  a  natural  way  of  implementing  this  dynamic 
binding. 
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OPTIC  FLOW  AND  THE  PERCEPTION  OF  SELF  MOTION  AND  RELATIVE  DEPTH 


MARTIN  S.  BANKS 
University  of  California  Berkeley 


Performance  varies  dramatically  with  retinal  eccentricity  in  a  variety  of  visual 
tasks.  In  the  first  part  of  the  presentation,  I  will  discuss  eccentricity-dependent 
variations  in  contrast  sensitivity,  spatial  summation,  and  grating  and  vernier  acuity. 
Then,  using  an  ideal  observer  analysis,  I  will  describe  the  extent  to  which  optics, 
photoreceptor  properties,  and  receptor  pooling  at  various  eccentricities  limit  such 
capabilities.  In  the  second  part  of  the  presentation,  I  will  discuss  eccentricity- 
dependent  variations  in  observers'  abilities  to  make  heading  judgments  from  optic 
flow  displays.  Performance  in  this  task  varies  much  less  with  eccentricity  than  in 
the  tasks  considered  in  the  first  part  of  the  presentation.  Finally,  I  will  describe  a 
simple,  biologically  plausible  model  of  how  heading  information  can  be  extracted 
from  optic  flow  displays. 
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CONSTRAINTS  ON  LEARNING  MAPPINGS  BETWEEN  SPATIAL  DIMENSIONS 

FELICE  L.  BEDFORD 
University  of  Arizona 


The  coordination  of  spatial  maps  from  different  modalities  is  essential  for  the 
experience  of  a  coherent  world  and  the  production  of  appropriate  behavior.  Little  is 
known,  however,  about  how  the  different  dimensions  arc  put  together.  Two  new 
variants  of  the  classic  prism-adaptation  paradigm  were  used  to  investigate  the  rules 
and  constraints  on  learning  new  mappings  between  visual  and  proprioceptive 
dimensions.  First,  rather  than  distort  a  large  continuous  region  of  space,  only  a  few 
isolated  locations  were  altered.  The  form  of  generalization  to  untrained  regions  was 
then  inspected  to  determine  the  hidden  rules  the  system  brings  to  bear.  Second, 
rather  than  a  simple  lateral  displacement,  more  complicated  new  mappings  were 
imposed  such  as  a  many-to-onc  mapping,  to  determine  behavior  in  a  variety  of 
situations.  A  new  technique  was  used  that  enabled  individual  regions  to  be  isolated 
and  manipulated  separately.  The  experiments  find  a  preference  for  linear  relations 
and  suggest,  tentatively,  that  the  hierarchy  of  geometric  properties  within  the 
framework  of  Euclidean  geometry  may  be  an  embodied  constraint  on  the  system  that 
realigns  maps. 
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THE  ROLE  OF  EFFERENCE  IN  SPACE  CONSTANCY 


BRUCE  BRIDGEMAN 
University  of  California 
Santa  Cruz 


Reflecting  distinctions  made  in  Icsioncd  animals  and  brain-damaged  humans,  two 
modes  of  visual  processing  can  be  identified  in  normal  humans.  A  "cognitive"  system 
is  assessed  with  perceptual  measures  requiring  a  symbolic  relationship  between 
target  location  and  behavioral  response.  A  second,  "sensorimotor”  system,  serving 

visually  guided  behavior,  is  assessed  with  open-loop  pointing  or  looking  measures 
involving  a  1:1  isomorphic  relationship  between  target  position  and  response. 

The  sensorimotor  system  requires  an  egocentric  calibration,  while  the  cognitive 
system  relies  largely  on  an  image-centered  metric.  Thus  the  role  of  oculomotor 

efference  can  be  expected  to  differ  in  the  two  systems.  Experiments  in  my  laboratory 

and  others  have  revealed  that  the  sensorimotor  system  relies  more  on  efference- 
copy  mechanisms  than  docs  the  cognitive  system.  Pressing  on  the  side  of  the  eye  to 
separate  gaze  position  from  efference.  we  have  shown  that  the  sensorimotor 

calibration  is  always  affected  by  efference  copy,  even  in  a  structured  visual  field, 

while  cognitive  judgments  arc  influenced  about  equally  by  efference  copy  and  by 
visual  context. 

Double  dissociation  experiments  have  changed  spatial  information 
independently  in  only  one  system,  or  only  the  other.  The  cognitive  system  is  subject 
to  many  spatial  illusions  that  have  little  effect  on  the  sensorimotor  system,  while  the 
latter's  memory  for  positions  of  objects  no  longer  present  is  limited. 
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THE  MAINTENANCE  OF  SPATIAL  ACCURACY  BY  VISUAL  NEURONS 
IN  THE  MONKEY  CEREBRAL  CORTEX 


MICHAEL  E.  GOLDBERG 
National  Eye  Institute 


Humans  (Hallett  and  Lightstone,  1976)  and  monkeys  (Mays  and  Sparks,  1980)  can 
make  accurate  saccadic  eye  movements  to  visual  targets  even  when  there  is  a 
dissociation  between  the  retinal  location  of  the  target  and  the  direction  of  the  eye 
movement  necessary  to  acquire  it.  Such  a  dissonance  is  established  by  a  double-step 
task  in  which  two  stimuli.  A  and  B.  are  flashed  sequentially  within  a  saccadic 
reaction  time  and  the  subject  is  obliged  to  make  successive  saccades  from  the  fixation 
point  to  A  and  then  from  A  to  B.  The  saccade  from  A  to  B  cannot  be  computed  from 
the  retinal  location  of  the  second  target,  but  requires  knowledge  either  of  the  target 
position  in  space  or  the  metrics  of  the  intervening  saccade. 

A  subset  of  visual  neurons  in  the  posterior  parietal  cortex  and  frontal  cortices  of 

the  rhesus  monkey  discharge  after  the  first  and  before  the  second  saccade  in  the 

double-step  task  when  this  second  saccade  matches  the  saccade  necessary  to  acquire  a 
stimulus  in  the  neuron's  receptive  field.  We  argue  that  this  discharge  represents  a 
transformation  of  the  visual  signal  from  retinal  to  motor  coordinates;  1)  The  activity 
in  the  double-stcp  task  cannot  be  a  premovement  discharge  since  these  neurons  do 
not  necessarily  discharge  when  the  monkey  makes  a  similar  learned  saccade  without 

a  visual  stimulus;  2)  The  discharge  cannot  be  postsaccadic  activity  related  to  the  first 
saccade  since  the  neurons  do  not  discharge  after  that  saccade  when  it  is  performed  in 
isolation;  3)  The  discharge  cannot  code  the  spatial  location  of  the  target  since  the 
neurons  do  not  discharge  when  the  monkey  makes  a  saccade  to  the  second  target 

directly  from  the  fixation  point.  We  conclude  that  this  activity  represents  a  visual 
response  to  a  stimulus  that  can  be  acquired  by  a  saccade  of  specific  direction  and 
amplitude  regardless  of  the  retinal  location  of  that  stimulus.  These  neurons  thus  have 

visual  receptive  fields  coded  in  motor  coordinates  but  do  not  explicitly  represent 
target  position  in  space. 

One  possible  explanation  for  this  phenomenon  is  that  the  transformation  from 

retinal  to  motor  coordinates  is  effected  by  a  discharge  corollary  to  the  intervening 

saccade  which  effectively  performs  a  vector  subtraction  of  the  dimensions  of  the 
first  saccade  from  the  retinal  location  of  the  target.  However,  analysis  of  error  trials 
in  double-step  tasks  (in  which  the  monkey  either  reverseu  the  order  of  the  saccades 

or  made  a  saccade  directly  to  the  second  target)  shows  that  the  cells  discharge  in 
response  to  the  second  target  whenever  it  appeared  in  the  context  of  the  double-stcp 
task,  and  not  only  when  the  monkey  actually  made  the  first  saccade.  Thus  the  neural 
processes  underlying  this  coordinate  transformation  have  access  to  processing  that 
precedes  the  aetual  performance  of  the  intervening  movement  and  arc  not 
dependent  upon  the  movement  actually  taking  place. 

This  transformation  provides  a  mechanism  whereby  the  oculomotor  system  can 
function  accurately  in  space  without  ever  requiring  the  representation,  by  a  single 
neuron  or  by  a  distributed  ensemble,  of  the  absolute  position  of  a  stimulus  in  a 
supraretinal  coordinate  system.  All  of  the  signals  necessary  to  implement  a  trans¬ 
formation  from  retinal  to  motor  coordinates  have  been  observed  in  the  frontal  and 
parietal  cortices.  This  is  especially  convenient  because  signals  in  brainstem 
structures  such  as  the  superior  colliculus  and  pontine  and  mesencephalic  burst 
regions  code  amplitude  and  direction  of  saccade  rather  than  target  position  in  space. 
Such  a  direct  transformation  obviates  the  need  for  an  explicit,  head-centered 
representation  of  space. 
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VISUAL-VESTIBULAR  INTERACTIONS  IN  PERCEIVED  STABILITY 


IAN  P.  HOWARD 
York  University 


The  veslibulo-ocular  reflex  (VOR)  serves  to  stabilize  the  retinal  image  when  the 
head  moves.  However  the  vestibular  organs  do  not  respond  to  low-frequency  or 
steady  motion  of  the  head.  Under  these  circumstances  the  visually-driven 
optokinetic  response  (OKN)  supplements  VOR.  The  sense  of  self  motion  must  also  rely 
on  visual  inputs  when  the  head  moves  slowly  or  steadily.  The  sense  of  self  motion 
engendered  by  visual  motion  is  known  as  vcction  and.  when  the  body  is  not  actually 
moving,  vcction  is  said  to  be  illusory.  We  have  been  investigating  how  visual- 
vestibular  interactions  that  influence  visual  stability  and  the  sense  of  self  motion 
have  become  adapted  to  the  demands  of  stereoscopic  vision.  We  have  shown  that  the 
conditions  that  evoke  OKN  are  very  different  from  those  that  evoke  vection,  even 
though  the  two  responses  arc  both  mediated  by  the  movement  of  large  visual  displays. 
OKN  is  evoked  only  by  moviig  displays  whic’.  lie  within  the  plane  of  binocular 
convergence.  This  mechanism  ensures  that  parallactic  motion  signals  arising  from 
outside  »hc  plane  of  convergence  do  not  disrupt  OKN  when  a  person  moves  linearly 
through  a  three-dimensional  world.  Vection,  on  the  other  hand,  is  triggered  by 
those  parts  of  a  visual  scene  which  arc,  or  arc  pciccivcd  to  be,  more  distant,  whether 
or  not  they  fall  within  the  plane  of  convergence.  This  makes  ecological  sense 

because  the  motion  of  more  distant  parts  of  a  natural  scene  is  a  more  reliable 
indicator  of  self  motion  than  is  that  of  nearer  parts  of  a  scene.  This  difference 

between  the  rules  governing  OKN  and  vcction  has  enabled  us  to  distinguish  between 

three  types  of  induced  visual  motion  (IM),  that  is,  illusory  visual  motion  of  a  test 
target  seen  against  a  moving  induction  display.  These  types  of  IM  form  a  nested 
hierarchy.  Oculoccntric  IM  occurs  within  a  retinal  frame  of  reference  and  is  due  to 

some  form  of  contrast  between  visual  motion  detectors.  Hcadccntric  IM  occurs 
within  a  hcadccntric  frame  of  reference  and  includes  effects  due  to  misregistration 
of  eye  movements.  Exoccntric  IM  occurs  within  an  external  frame  of  reference  and 
includes  the  apparent  motion  of  stationary  objects  that  accompanies  illusory  self 
motion.  We  have  found  tests  for  each  of  these  forms  of  IM  and  have  revealed  their 

respective  magnitudes, 
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EFFECTS  OF  CHANGING  TARGET  LOCATION  ON  PREHENSION  MOVEMENTS 


MARC  JEANNEROD 
Bron,  France 


Prehension  involves  processing  information  in  two  hypothesized  visuomotor 
channels:  one  for  extrinsic  object  properties  (e.g..  the  spatial  location  of  objects)  and 

one  for  intrinsic  objects  (e.g.,  shape  and  size).  The  present  study  asked  how  the  two 
motor  components  that  correspond  to  these  channels  (transport  and  grasp, 
respectively)  are  related.  One  way  to  address  this  question  is  to  create  a  situation 
where  unexpected  changes  occur  at  the  input  level  of  one  of  the  visuomotor 
channels,  and  to  observe  how  the  movement  reorganizes.  If  transport  and  grasp  are 
independent  components,  then  changing  the  object  location,  for  example,  should 
affect  only  the  transport,  not  the  grasp  component.  Subjects  were  requested  to 
reach,  grasp  and  lift  as  accu  ly  as  possible  one  of  three  dowels  using  the  distal 
pads  of  the  thumb  and  index  linger.  On  certain  trials,  upon  movement  initiation 
towards  the  middle  dowel,  the  dowel  was  made  to  instantanteously  change  its  location 
to  one  of  the  two  other  positions,  requiring  the  subjeet  to  reorient  the  hand  to  the 
new  dowel  location.  Results  consisted  of  comparing  the  movement  characteristics  of 
the  transport  and  grasp  components  of  these  perturbed  movements  with  appropriate 
control  movements. 

Kinematics  of  the  wrist  trajectory  showed  fast  adjustments,  within  100  ms,  to  the 
change  of  dowel  position.  This  duration  seems  to  correspond  to  the  minimum  delay 
required  within  the  visuomotor  .system  for  visual  and/or  proprioceptive  reafferents 
to  influence  the  ongoing  movement.  In  addition,  these  delays  are  much  shorter  than 
has  been  found  for  conditions  where  object  location  changes  before  movement 

initiation  (approximately  300  ms).  The  faster  times  may  relate  to  the  dynamic 
character  of  the  deviant  limb  position  signals,  with  the  only  constraint  being  the 
physiological  delays  for  visual  and  kinesthetic  signals  to  influence  the  movement. 

A  spatiotemporal  variability  analysis  of  the  movement  trajectories  for  non- 
perturbed  trials  showed  variability  to  be  greatest  during  the  acceleration  part  of  the 
movement,  interpreted  as  due  to  control  by  a  relatively  inaccurate  directional  coding 
mechanism.  Control  during  the  deceleration  phase,  marked  by  low  trajectory 
variability,  was  seen  to  be  due  to  a  sensorimotor  process,  using  motor  output  signals, 
and  resulting  in  an  optimized  trajectory  supporting  a  successful  grasp. 

Analysis  of  the  grasp  component  of  prehension  showed  that  perturbing  object 

location  influenced  the  movement  of  the  fingers  suggesting  a  kinematic  coupling  of 
the  two  components.  However,  forthcoming  work  shows  that,  when  object  size 

changes,  and  location  remains  constant,  there  is  a  clear  temporal  dissociation  of  the 
two  components  of  prehension.  Collectively,  these  results  suggest  that  the  two 
visuomotor  channels  have  different  'ime  constraints  with  the  time-constant  of  the 

channel  activated  by  the  perturbation  constraining  the  timing  of  the  other. 
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EFFERENCE  COPY  IN  THE  CONTROL  OF  THE  VESTIBULO-OCULAR  REFLEX 


W.  MICHAEL  KING  &  LAWRENCE  H.  SNYDER 
University  of  Rochester 


Many  cxpcrimenis  suggest  that  the  brain  uses  "corollary  discharge"  or 

"effercncc  copy”  signals  of  eye  position  to  localize  objects  with  respect  to  the  body  or 
to  direct  eye  movements.  The  origin  and  nature  of  these  signals  is  usually  not 
specified.  For  example,  the  vcstibulo-ocular  reflex  (VOR),  which  stabilizes  retinal 
images  during  head  movements,  is  modulated  inversely  with  viewing  distance.  Since 

vergence  angle  is  also  modulated  inversely  with  viewing  distance,  it  has  been 
suggested  that  "vergence  stale”  is  used  to  modify  the  VOR.  We  compared  the  temporal 
relationship  between  changes  in  vergence  and  changes  in  the  VOR  to  help 

characterize  the  source  of  such  a  signal.  On  the  one  hand,  if  a  corollary  discharge  of 
the  motor  output  of  the  vergence  system  modulates  the  VOR,  then  we  would  expect 
that  changes  in  the  VOR  would  be  coincident  with  changes  in  vergence  angle.  On 
the  other  hand,  if  sensory  cues  encoding  target  distance  modulate  the  VOR,  then  we 
would  expect  that  changes  in  the  VOR  would  lead  changes  in  vergence  angle. 

Four  rhesus  monkeys,  each  equipped  with  two  scleral  search  coils  to  monitor  the 

movements  of  both  eyes,  were  trained  to  fixate  LED  targets.  Trials  began  with 
fixation  of  a  target  13.5,  18  or  36  cm  from  the  eyes.  After  at  least  0.6  s  of  fixation,  a 
vergence  movement  was  evoked  by  lighting  a  near  (9  cm)  or  far  (220  cm)  LED. 
Randomly,  before,  during  or  after  the  vergence  movement,  the  monkey  was  briefly 
rotated  (0.5  d/s/ms  acceleration  to  20  d/s)  to  the  right  or  left.  This  pulse  of  head 
velocity  provided  a  near-instantaneous  assay  of  VOR  response. 

The  amplitude  of  the  VOR  changed  systematically  during  the  course  of  vergence 
eye  movements.  These  changes  anticipated  ehanges  in  vergence  by  as  much  as  100 
ms,  suggesting  that  sensory  cues  encoding  target  distance  drive  VOR  modulation. 
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LOTZE'S  LOCAL  SIGN:  A  MECHANISM  FOR  JUDGING  DISTANCE 


DENNIS  M.  LEVI 
University  of  Houston 


"The  space  relations  of  impressions  pass  as  such  into  the  soul,  ...each  impression 
is  conveyed  to  it  by  a  distinct  fibre,  and  the  fibres  reach  the  scat  of  the  soul  with 
their  relative  situation  wholly  undisturbed."  H.  R.  Lotze  (1885). 

In  order  to  properly  order  stimuli  in  visual  space,  and  to  program  saccadic  eye 
movements,  H.R.  Lotze  proposed  that  each  retinal  element  has  a  position  label,  or 

local  sign.  Evidence  will  be  presented  to  suggest  that  in  peripheral  vision,  a  point 

can  be  localized  with  a  precision  better  than  1%  of  the  target  eccentricity.  This 
precision  is  similar  to  the  separation  of  on-pBcta  ganglion  cells  in  the  retinal 
periphery  (Wassle  et  al.,  1990),  suggesting  that  the  position  of  each  peripheral 
ganglion  cell  is  indeed  labeled. 

The  notion  of  local  signs  has  important  consequences  for  the  encoding  of 

position.  In  position  acuity  tasks  (e.g..  Vernier  acuity  and  spatial  interval 
discrimination),  over  a  wide  range  of  target  separations,  thresholds  are  proportional 
to  the  separation  of  the  features.  This  is  Weber's  law  for  position.  We  argue  that  two 
distinct  processes  constrain  position  acuity,  and  each  leads  to  Weber's  law:  i)  when 

the  target  separation  is  small,  position  acuity  is  limited  by  the  output  of  size  tuned 
filters.  Weber's  law  obtains  as  a  consequence  of  using  larger  filters  for  targets  with 
larger  separations.  The  outputs  of  these  localized  filters  can  give  rise  to  the  fine 
position  thresholds  known  as  hypcracuily  thresholds.  ii)  when  the  target  separation 
is  large,  the  eccentricity  of  the  targets,  i.e.,  the  local  signs,  constrain  position 
discrimination.  Position  discrimination  experiments  on  iso-eccentric  arcs  (where 
separation  and  eccentricity  can  be  pitted  against  each  other)  suggest  that  both 
processes  constrain  position  judgments. 

The  notion  of  local  signs  also  has  important  consequences  for  programming  of 
saccadic  eye-movements.  Although  it  is  widely  held  that  the  precision  of  saccadic 
eye  movements  is  limited  by  the  motor  system,  recent  eye  movement  data  (van  Opstal 
and  van  Gisbergen,  1989),  suggests  that  the  precision  of  saccadic  eye  movements  to 
peripheral  targets  can  be  almost  as  precise  as  the  localization  of  a  point  in  the 
periphery.  Thus,  the  precision  of  saccadic  eye  movements  may  be  primarily  limited 
by  the  sensory  limitations  in  encoding  the  position  of  peripheral  targets,  i.e.,  by 
Lotze's  local  sign. 


VISUAL  MOTION  SIGNALS  SUPPORTING  STABILIZATION  OF  GAZE  BY 

PURSUIT  EYE  MOVEMENTS 

STEPHEN  G.  LISBERGER.  J.  ANTHONY  MOVSHON,  RICHARD  J.  KRAUZLIS 
University  of  California  at  San  Francisco 


Eye  movements  stabilize  the  images  of  moving  objects  by  rotating  the  eyes  at  the 
velocity  of  the  object.  It  has  long  been  known  that  visual  motion  drives  pursuit,  and 
recent  anatomical  and  neuropsychological  studies  suggest  that  visual  motion  signals 
carried  by  neurons  in  area  MT  (V5)  of  the  macaque's  extrastriate  cortex  are  critical 
for  the  generation  of  pursuit.  The  goal  of  our  study  was  to  reveal  the  dynamic 
properties  of  signals  that  are  actually  used  for  pursuit  and  to  determine  whether 
those  signals  are  available  in  MT. 

In  one  series  of  experiments,  we  recorded  the  pursuit  eye  movements  of  a  trained 
rhesus  monkey  in  response  to  the  motion  of  a  small  target.  Stimuli  were  steps  or 
ramps  of  target  velocity,  which  provided  stimuli  moving  at  constant  speed  or 
constant  acceleration.  We  analyzed  only  the  first  100  ms  of  pursuit  so  that  we  use  the 
open-loop  response  of  the  pursuit  system  as  a  probe  for  the  properties  of  visual 
signals  that  drive  pursuit.  Steps  of  target  velocity  elicited  eye  acceleration  that  could 
be  dissected  into  two  components.  An  early  "motion  onset  transient"  dominated  the 
first  40  ms  of  pursuit  and  was  driven  by  the  impulse  of  target  acceleration  at  the 
onset  of  motion.  A  later  steady-state  response  had  a  magnitude  that  was  linearly 
related  to  target  speed.  The  motion  onset  transient  was  affected  by  varying  the 
motion  onset  delay,  defined  as  the  period  when  the  tracking  target  was  stationary 
and  visible  before  it  started  moving.  It  was  not  evident  if  the  target  was  moving 
when  it  came  on,  but  became  larger  as  the  motion  onset  delay  was  increased  to  200  ms. 
Ramps  of  target  velocity  revealed  a  third  component  of  eye  accelcraton  that  was 
related  to  smooth  target  acceleration  and  could  not  be  accounted  for  by  either  the 
motion  onset  transient  or  the  steady-state  target  velocity  component. 

In  a  second  series  of  experiments  we  recorded  single  MT  units  in  anesthesized, 
paralyzed  Cynomolgus  monkeys,  and  analyzed  the  responses  elicited  by  moving 
texture  targets  of  variable  speed  and  direction  of  motion.  To  analyze  response 
dynamics,  we  modulated  target  speed  with  suitable  pulses,  steps,  and  ramps.  We  paid 
particular  attention  to  the  time  course  of  the  responses  of  MT  neurons  so  that  we 
could  compare  the  signals  they  carry  to  those  used  by  pursuit.  As  we  expected,  almost 
all  MT  units  were  selective  for  the  direction  and  speed  of  stimulus  motion.  Most  units 
also  gave  brisk  responses  to  target  acceleration.  Analysis  of  these  resonses  reveals 

that  they  result  from  the  action  of  two  mechanisms  within  MT  receptive  fields.  The 
first  is  the  spatio-temporal  tuning  of  the  unit,  which  gives  it  speed  selectivity.  When 
stimulus  speed  increases  to  and  then  past  the  preferred  speed,  the  resulting 
excitation  profile  is  a  shaped  pulse,  causing  a  transient  response  during  acceleraton. 
The  second  mechanism  enhances  responses  to  changes  in  excitation  level,  such  as 
those  occurring  during  target  acceleration.  This  mechanism  can  be  modeled  as  a 
filter  shaping  the  time  course  of  response  to  a  degree  that  varies  among  units.  These 
two  mechanisms  combine  to  generate  accceleration-rclated  responses  in  MT  neurons, 
without  explicitly  creating  acceleration-tuned  receptive  fields.  The  two  mechanisms 
also  cause  MT  cells  to  emit  a  motion  onset  transient  followed  by  sustained  firing  in 
response  to  steps  of  target  speed.  The  amplitude  of  the  motion  onset  transient  of  MT 
cells,  like  that  in  pursuit,  increases  as  motion  onset  delay  is  increased.  Our  results 
show  that  MT  neurons  carry  motion  signals  that  can  explain  the  contributions  of 
target  speed  and  acceleration  to  the  generation  of  pursuit  eye  movements. 


CALIBRATION  OF  SIMPLE  VISUAL  SYSTEMS 
BY  COMPARISON  ACROSS  EYE  MOVEMENTS 

LAURENCE  T.  MALONEY 
New  York  University 


A  visual  system  is  geometrically  calibrated  if  its  estimates  of  the  spatial 
properties  of  a  scene  are  accurate:  straight  lines  arc  judged  straight,  angles  are 
correctly  estimated,  collincar  line  segments  are  perceived  to  fall  on  a  common  line.  I 

describe  calibration  methods  for  a  model  visual  system  whose  photoreceptors  are 
initially  at  unknown  locations.  The  methods  can  also  compensate  for  optical 
distortions  that  arc  equivalent  to  remapping  of  receptor  locations  (c.g.,  spherical 
aberration).  The  methods  work  by  comparing  visual  input  across  eye/head 
movements;  they  require  no  explicit  feedback,  and  no  knowledge  about  the 
particular  contents  of  a  scene. 

It's  likely  that  no  biological  visual  system  is  ever  perfectly  calibrated,  but 

considerable  evidence  exists  that  biological  visual  systems  do  compensate  for  optical 
distortions  and  initial  uncertainty  about  the  position  of  photoreceptors  in  the  retinal 
photoreceptor  lattice.  Recent  anatomical  work,  for  example,  demonstrates  apparent 
disorder  in  the  retinal  lattice  outside  central  fovea,  increasing  with  eccentricity. 
Further,  the  optics  of  the  eye  change  throughout  the  lifespan,  suggesting  that 

calibration  may  continue  in  the  adult. 

Previous  work  in  visual  neural  development  suggests  a  variety  of  sources  of 
information  that  drive  calibration,  and  there  are  computational  models  of  visual 

neural  development  based  on  these  cues.  Yet,  although  biological  visual  systems  are 
known  to  require  patterned  visual  stimulation  to  achieve  normal  organization,  none 
of  these  models  requires  such  stimulation  to  function.  Further,  while  these  models 
could  in  principle  compensate  for  disorder  in  the  retinal  lattice,  none  of  them 
clearly  addresses  the  problem  of  compensation  for  optical  distortion. 

I  describe  two  methods  for  calibrating  a  simple  linear  visual  system  that  work  by 
comparing  visual  input  across  eye/head  movements.  These  methods  can  organize  the 
receptive  fields  of  a  simple  visual  system  so  as  to  compensate  for  irrregularities  in 
the  retinal  photoreceptor  lattice  and  optical  irregularities  equivalent  to  distortions 
in  the  lattice.  They  require  no  cxplict  feedback,  and  no  knowledge  about  the 
particular  contents  of  a  st.iie,  but  instead  work  by  'asserting'  that  the  internal 
representation  of  the  scene  behaves  in  a  prespecified  way  under  eye  and  head 
movements.  I  demonstrate  that  these  methods  can  be  used  to  calibrate  a  simple, 
linear  visual  system  and  I  discuss  the  implications  of  this  work  for  other  models  of 
visual  processing. 


SURFACE  PERCEPTION  FROM  DISPARITY  AND  PARALLAX  INFORMATION 

BRIAN  J.  ROGERS 
University  of  Oxford 


Most  work  on  binocular  stcrcopsis  (psychophysical,  computational  and 
physiological)  has  assumed  that  the  intitial  representation  of  disparity  is  in  terms  of 
the  horizontal  (and  possibly  the  vertical)  differences  between  corresponding  points 
in  the  images  (e.g.  Julcsz,  1971;  p.  144).  An  alternative  conception,  which  brings  out 
the  formal  similarity  between  motion  parallax  and  binocular  disparity  as  sources  of  3- 
D  information,  is  to  consider  the  local  differential  structure  in  the  transformation 
needed  to  map  one  eye's  image  onto  the  other  (Koenderink  and  van  Doom,  1976).  In 
particular,  the  amount  of  deformation  between  corresponding  regions  in  the  two 
binocular  images  (or  the  amount  of  deformation  over  time  in  the  case  of  the  parallax 
transformation)  provides  information  about  the  local  slant  of  3-D  surfaces.  The 
existence  of  (i)  a  similar  and  marked  anisotropy  in  the  perception  of  3-D  surfaces 
from  either  disparity  or  parallax  information  (Rogers  and  Graham,  1983)  and  (ii)  an 
analogous  "induced  effect"  (Ogle,  1950)  in  the  parallax  domain  (Rogers  and 
Koenderink,  19986)  are  both  consistent  with  this  alternative  conceptualization. 
Furthermore,  Cagenello  and  Rogers  (1988.  1989)  have  shown  that  the  orientation  of 
line  elements  superimposed  on  stereoscopic  surfaces  has  a  significant  effect  on  their 
perceived  slant  and  curvature.  The  latter  result  is  consistent  with  Koenderink's 
suggestion  that  an  approximation  to  deformation  could  be  computed  by  monitoring 
the  orientation  differences  between  corresponding  line  elements  at  two  or  more 
orientations  in  the  two  eyes'  images  (in  the  case  of  binocular  stereopsis)  or  the 
changes  of  orientation  of  local  line  elements  occurring  over  lime  (in  the  case  of 
motion  parallax). 

Although  attractive  and  consistent  with  much  of  the  empirical  evidence,  the 
deformation  hypothesis  docs  not  account  for  several  recent  observations.  First, 
stereoscopic  images  related  by  a  vertical  shear  transformation  are  seen  to  have  the 
opposite  direction  of  slant  to  that  specified  by  the  local  deformation.  Second, 
stereoscopic  images  related  by  a  pure  rotation  generate  the  impression  of  surface 
slant  with  a  magnitude  comparable  to  that  produced  by  images  related  by  a  simple 
horizontal  shear,  even  though  there  is  no  deformation  between  the  two  images. 

These  results  suggest  that  the  human  visual  system  might  rely  on  the  simpler 
strategy  of  extracting  the  orientation  differences  between  corresponding  binocular 
elements,  rather  than  computing  the  true  differential  structure  of  disparity  and  flow 
fields. 


MOTOR  CORTICAL  CONTRIBUTIONS  TO  SKILLED  MOVEMENT 


ANDREW  B.  SCHWARTZ 
Barrow  Neurological  Institute 


Unitary  activity  recorded  in  the  proximal-arm  area  of  primary  motor  cortex 
activity  is  sensitive  to  the  direction  that  the  contralateral  arm  moves.  The  tuning 
function  that  relates  activity  to  direction  is  broadly  tuned  for  many  cells,  spanning 
the  entire  directional  domain,  with  the  peak  firing  rate  in  a  single  "preferred 
direction."  This  suggests  that  these  cells  arc  all  coding  movement  direction 
simultaneously  during  movement.  A  vector  in  each  cell's  preferred  direction  was 
scaled  by  the  cell’s  discharge  rate  to  represent  its  contribution  to  the  population.  The 
response  of  the  entire  population  of  these  cells  recorded  during  several  behavioral 
tasks  can  be  represented  by  adding  vectors  in  each  cell's  preferred  direction.  The 
"population  vectors,"  resulting  from  the  addition  of  these  individual  vectors,  point  in 
the  direction  of  movement. 

Rhesus  monkeys  were  trained  to  trace  with  their  index  fingers  sine  waves  of 
different  amplitudes  and  spatial  frequencies  generated  on  a  computer  monitor. 

These  animals  learned  the  task  by  following  a  circle  as  it  moved  along  a  randomly 
selected  sinusoidal  pattern.  By  adjusting  the  size  of  the  circle  and  the  maximum  time 
allowed  to  reach  the  circle,  the  monkeys  were  conditioned  (with  a  liquid  reward)  to 
make  smooth  and  accurate  movements. 

Motor  cortical  activity  recorded  during  this  task  was  dependent  on  the  direction 
the  monkey  was  to  move  its  finger  100-150  msec  later.  Population  vectors  (using  the 
responses  of  238  cells)  calculated  throughout  the  task  (n=100)  pointed  along  the 
tangent  of  the  corresponding  part  of  the  trajectory.  Since  the  direction  of  the 
population  vector  corresponded  to  the  tangent  of  the  movement,  it  seems  likely  that 
the  motor  cortex  is  using  a  relative  coordinate  system  representing  instantaneous 
direction.  The  length  of  each  population  vector  calculated  in  the  task  corresponded 
to  the  speed  of  movement,  again  suggesting  that  these  skilled  movements  are 
represented  incrementally  in  the  activity  of  motor  cortical  nuerons.  This  work  is 
supported  by  NIH  (NS-26375). 


HUMAN  EYE  MUSCLE  PROPRIOCEPTION: 
ITS  EXISTENCE  AND  ITS  RAISON  D*  ETRE 


MARTIN  J.  STEINBACH 
Hospital  for  Sick  Children 
Eye  Research  Institute  of  Ontario 


We  have  measured  changes  in  eye-hand  coordination  following  surgery  on,  or 
pharmacological  weakening  of,  eye  muscles.  By  preventing  vision  in  the  operated 

eye  until  the  moment  of  testing,  or  in  ruling  out  efferent  (corollary  discharge) 
contributions  to  eye  position  sensing,  we  have  found  evidence  for  the  use  of  an 
afferent  (proprioceptive)  signal  in  determining  the  position  of  the  eye  in  the  orbit. 

Our  latest  studies  of  resting  eye  position  in  anesthetized  and  paralyzed  patients 
have  shown  that  the  direction  of  gravity  influences  this  resting  position.  Our 
findings  indicate  that  the  center  of  rotation  of  the  eye  is  actually  in  front  of  its 
center  of  mass.  Eye  muscles,  therefore,  operate  under  changing  loads  as  the  head 
changes  its  orientation  with  respect  to  gravity  and  thus  are  like  other  skeletal 
muscles  in  having  a  need  for  proprioceptors.  Our  finding  may  also  be  playing  a  role 
in  the  space  motion  sickness  that  some  astronauts  experience  in  microgravity. 

Finally,  the  vexing  species  differences  concerning  the  presence  of  eye  muscle 
spindles  may  be  related  to  the  need  for  them,  i.e.,  animals  with  no  spindles  may  have 
eyes  with  coincident  centers  of  rotation  and  mass. 


NAVIGATION  FROM  OPTICAL  FLOW 


WILLIAM  H.  WARREN 
Brown  University 


Gibson  (1950)  first  proposed  that  patterns  of  optical  flow  produced  at  a  moving 
eye  specify  the  observer's  direction  of  self-motion,  or  heading.  Such  information  is 
critical  for  navigation  in  complex  environments.  In  this  vein,  I  argue  that  different 
flow  morphologies  specify  different  classes  of  observer  movement,  including  linear 
translation,  circular  movement,  combined  translation  and  rotation  (as  with  pursuit 
eye  movements  during  locomotion),  and  combined  circular  movement  and  rotation. 
Recent  experiments  with  random-dot  displays,  in  which  observers  judged  whether 
they  would  pass  to  the  right  or  left  of  a  target,  demonstrate  that  the  visual  system  can 
use  such  information  to  determine  heading. 

(1)  Translation:  Observers  perceive  translational  heading  with  an  accuracy  of  1 
deg,  in  both  planar  and  3D  environments.  There  are  no  effects  of  dot  density  down  to 
3  dots,  and  a  2-frame  dot  life  is  sufficient.  This  indicates  that  judgments  are  based  on 
the  radial  structure  of  the  velocity  field,  rather  than  a  local  focus  of  expansion.  In 
addition,  the  visual  system  can  tolerate  large  amounts  of  noise  in  local  vector 
direction  or  magnitude,  indicating  spatial  pooling  over  the  field.  A  neural  network 
model  with  these  characteristics  is  discussed. 

(2)  Circular  movement:  Observers  perceive  their  future  curved  path  with  an 
accuracy  of  1.5  deg  (for  a  target  16m  ahead),  in  both  planar  and  3D  environments. 
There  are  no  effects  of  dot  density  down  to  2  dots,  but  a  3-frame  dot  life  is  necessary; 
the  instantaneous  velocity  field  is  both  theoretically  and  empirically  insufficient. 

This  is  consistent  with  an  analysis  showing  that  the  change  in  direction  of  motion  of 
2  elements  over  3  frames  is  sufficient  to  specify  circular  heading. 

(3)  Translation  and  rotation:  Observers  can  judge  translational  heading  in 
displays  that  simulate  the  optical  effects  of  a  combined  translation  and  eye  rotation 
with  an  accuracy  of  1.5  deg.  This  indicates  that  translational  and  rotational 
components  can  be  decomposed  on  the  basis  of  optical  flow  alone,  without  the 
contribution  of  an  extra-retinal  signal.  Further,  the  decomposition  is  possible  with  a 
ground  surface  and  a  3D  cloud,  but  not  with  movement  toward  a  wall  surface.  This 
pattern  of  results  is  inconsistent  with  general  discrete  models,  least-squares  models, 
and  differential  models  (such  as  div  and  curl)  that  assume  a  continuous  field,  but 
consistent  with  a  theory  based  on  relative  motion  between  neighboring  elements  at 
different  depths.  A  neural  network  model  that  exhibits  similar  behavior  is  discussed. 

The  resulting  heading  direction  is  given  in  both  retinal  coordinates  (anatomical 
position  on  the  retina)  and  environmental  coordinates  (ordinal  position  relative  to 
environmental  surfaces).  It  could  also  be  used  to  steer  locomotion,  for  a  change  in 
heading  direction  (desired  heading  -  current  heading)  could  be  made  by  effecting  an 
equivalent  change  in  the  direction  of  shear  force  against  the  ground  (desired  force  - 
current  force)  without  transformation  to  body  coordinates.  This  avoids  the  problem 
of  local  sign  and  coordinate  transformations  in  a  multi-degree-of-freedom  system. 


"SEE  AS  THOU  WAST  WANT  TO  SEE" 

AFTEREFFECTS  AND  THE  RECALIBRATION  OF  VISUAL  PERCEPTS 

JEREMY  M.  WOLFE 

Massachusetts  Institute  of  Technology 


Many  visual  processes  in  the  central  nervous  system  attempt  to  infer  properties 
of  .  the  external  world  on  the  basis  of  input  from  earlier  neural  stages.  For  example, 
an  opponent  color  mechanism  might  infer  the  amount  of  redness  or  greenness  by 
comparing  the  inputs  from  L  and  M  cone  pathways.  Variations  in  the  input  cause 

variations  in  the  inference.  There  arc  two  sources  of  variation  in  neural  input:  1)  A 

change  in  the  distal  stimulus  and  2)  a  change  internal  to  the  visual  system.  A  visual 
process  is  interested  in  the  first  source  of  variation  and  misled  by  the  second.  Thus, 
if  the  L  cone  input  is  somehow  damaged  and  weakened,  the  opponent  mechanism  will 
interpret  the  preponderance  of  M-conc  input  as  an  indication  of  "greenness." 

Since  vegetative  changes  in  the  visual  system  are  unavoidable  and  since  the 
consequences  are  unacceptable,  the  visual  system  has  developed  measures  to  correct 
for  internal  errors.  This  is  not  an  altogether  trivial  matter.  A  visual  process  has 
access  only  to  visual  information.  It  must  decide  if  that  information  reflects  the  state 
of  the  external  world  or  an  internal  problem  requiring  correction.  An  opponent 

color  mechanism  must  be  able  to  differentiate  between  a  visual  system  that  is  looking 

at  the  grass  while  cutting  the  lawn  and  a  system  suffering  a  failure  in  the  L-cone 

pathway . 

Traditional  models  of  error-correcting  mechanisms  do  not  make  this  distinction. 
They  will  correct  for  systematic  bias  in  the  input  but  arc  indifferent  to  the  cause  of 
that  bias.  The  thesis  of  this  talk  is  that  the  human  visual  system  is  not  indifferent  to 
this  distinction.  Internal  and  external  sources  of  variation  will  produce  different 
changes  in  the  neural  input.  Components  of  the  visual  system  are  designed  to  self- 
correct  in  response  to  internal  sources.  If  external  stimuli  are  designed  to  mimic  an 
internal  source  of  variation,  the  result  is  a  long-term  aftereffect.  For  example,  color 
and  orientation  are  generally  uncorrclatcd  in  visual  stimuli.  A  correlation  might  be 
a  sign  of  internal  error.  If  we  create  a  correlation  by  presenting  red  vertical  and 

green  horizontal  gratings,  the  result  is  a  long  term  aftereffect  known  as  the 
McCollough  effect.  It  can  last  for  days  following  minutes  of  adaptation.  If  subjects 
are  shown  red  vertical  and  red  horizontal  gratings,  then  there  is  no  correlation 
between  color  and  orientation  and,  even  though  either  stimulus  alone  will  produce 
an  effect,  little  or  no  aftereffect  is  seen. 

Different  visual  processes  have  different  filters  to  distinguish  internal  from 
external  variation.  For  example,  there  are  both  short  and  long  term  tilt  aftereffects 
(TAE).  Four  minutes  of  sustained  tilt  adaptation  will  produce  about  four  minutes  of 
TAE.  Four  minutes  of  intermittent  adaptation  can  produce  a  TAE  lasting  weeks.  In 
this  case,  we  believe  that  a  single,  unusual  stimulus  is  not  a  sign  of  internal  change. 
However,  if  many  samples  show  a  systematic  bias  as  is  the  case  in  intermittent 
stimulation,  the  system  can  be  fooled  into  recalibration. 

In  sum,  we  believe  that  the  components  of  the  visual  system  adjust  themselves  to 
correct  for  changes  internal  to  the  visual  system.  They  protect  themselves  from 
wanton  recalibration  by  placing  a  filter  between  the  input  and  the  adjustment 
mechanism.  The  filter  is  designed  to  pass  only  stimuli  that  are  unlikely  to  occur  in 
normal  viewing  and  may  be  different  for  different  components  of  the  visual  system. 


SPATIAL  ORIENTATION  IN  WEIGHTLESSNESS 
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Although  fundamental  visual  acuity  appears  unaffected  by  weightlessness  and 
the  most  common  visual  illusions  from  space  are  easily  explainable,  there  remain 
several  interesting  problems  in  spatial  orientation  in  the  absence  of  a  gravity 
reference  direction.  This  paper  touches  on  three  aspects  of  the  question: 

(1)  Dynamic  spatial  orientation,  based  upon  the  interpretation  of  information 
from  moving  visual  scenes  (vcction),  vestibular  stimuli  and  localized  tactile  inputs,  is 
dramatically  changed  in  orbit  as  the  irrelevant  otolith  cues  are  ignored  and  visual  as 
well  as  tactile  cues  begin  to  dominate. 

(2)  Static  orientation  can  produce  some  remarkable  illusions  in  the  absence  of  a 
gravity  reference  vector,  especially  when  the  astronaut  is  confronted  with  familiar 
objects  (the  equipment  racks,  another  crewmember,  the  earth)  in  unusual 
orientations.  The  "inversion  illusion"  and  others  arc  discussed  and  related  to  space 
motion  sickness. 

(3)  Tclcopcration  involves  a  complex  spatial  transformation  from  the  TV  scene 
taken  by  a  camera  on  a  remote  manipulator  to  the  fixed  display  in  front  of  the 
astronaut.  Special  issues  are  raised  when  the  sensory-motor  adaptation  process  must 
take  place  in  the  absence  of  a  reference  gravity  vector. 


